Photogrammetric methods and devices related thereto

ABSTRACT

The inventions herein relate generally to improvements in photogrammetry and devices suitable for obtaining such improvements. Some embodiments use only a single passive image-capture device to obtain overlapping 2D images, where such images at least partially overlap with regard to at least one object of interest in a scene. Such images can be processed using methods incorporating structure from motion algorithms. Accurate 3D digital representations of the at least one object of interest can be obtained. Substantially accurate measurements and other useful information regarding the at least one object of interest are obtainable from the methodology herein.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/066,925, filed Oct. 22, 2014, entitled “Photogrammetric Methods andDevices Related Thereto,” the disclosure of which is incorporated in itsentirety by this reference.

FIELD OF THE INVENTION

The inventions herein relate generally to improvements in photogrammetryand devices suitable for obtaining such improvements.

BACKGROUND OF THE INVENTION

Photogrammetry is the science of obtaining measurements fromphotographs, especially for recovering the exact or nearly-exactpositions of surface points. While photogrammetry is emerging as arobust, non-contact technique to obtain measurements of objects, scenes,landscapes, etc., there are limitations to existing methods, some ofwhich, for example, are set forth in the following few paragraphs.

Accurate three-dimensional (3D) digital representations of objects canbe obtained using methods that utilize active-sensing techniques, suchas systems that emit structured light, laser beams or the like, recordimages of objects illuminated by the emitted light, and then determinethe 3D measurements from the recorded images. A laser scanner is anexample of a standalone device that utilizes structured light togenerate measurements of objects. When used in mobile devices, such assmartphones and tablets, emission of the structured light used for 2Dand 3D image generation can be achieved by including a separate hardwaredevice as a peripheral. This peripheral is configured to emit, forexample, structured light to generate a point cloud (or depth map) fromwhich data about the object of interest can be derived usingphotogrammetric algorithms. Use of such a peripheral device to provideactive sensing methods are provided by, for example, Structure Sensor(see the internet URL structure.io), the DPI-8 kit or the DPI-8SR kitproducts (see the internet URL www.dotproduct3d.com). While oftenproviding accurate image data, it is nonetheless cumbersome for users tohave to add a clamp-on or other type of peripheral equipment to theirmobile devices. Alternatively, active sensing means can be integratedinto mobile devices, such as in Google's Tango® product.

Existing passive photogrammetry methods—that is, methods that do not usestructured light, lasers or the like but which, for example, utilizeimages captured by a camera from which to derive measurements, etc. canalso be problematic to use. Conventional stereo/2D or 3D camerastypically obtain two images of an object simultaneously from twoviewpoints that are typically separated, for example, by theinterpupillary distance (IPD) of a person (which can range from about 52to about 78 mm according to the 1988 Gordan et al. “AnthropometricSurvey of US Army Personnel, Methods and Summary Statistics.” TR-89-044.Natick Mass.: U.S. Army Natick Research, Development and EngineeringCenter). Such stereo images generally have insufficient parallax forhigh-quality measurement when used to obtain data regarding distantobjects (e.g., objects more than a few (about one to about five) metersaway from the cameras). To obtain suitable parallax using such methods,the user will be directed to use a template or framework incorporatedin, for example, software associated with the image-capture device toguide orientation of the image-capture device relative to the object ofinterest. This technique can ensure that a sufficient number ofappropriately overlapping images of the object of interest are obtained.Alternatively, the user can be provided with general instructions of howto orient the camera and/or object so as to obtain appropriate overlap.Both of these techniques for guiding the user can be used to provideaccurate visualization of the object of interest but are nonethelesscumbersome and prone to user error.

It is possible to obtain accurate measurements from photographs by usingmultiple images of an object of interest. When placed in a 3D context(i.e., “multiple view geometry”), the three-dimensional points from anobject of interest can be estimated from measurements from two or morephotographic images taken from different positions. Corresponding pointsare identified on each image. A line of sight (or ray) can beconstructed from the camera location to the point on the object.Triangulation allows determination of the 3D location of the point bothin relation to the object's orientation in space, as well as with regardto that point's orientation and/or position in relation to other points.

Methods for passive photogrammetry where 3D digital representations ofthe object(s) of interest can be used to derive measurements and otherdetail of interest are disclosed in U.S. Pat. No. 8,897,539 titled“Using images to create measurements of structures through thevideogrammetric process” and PCT Publication No. WO2013/173383 byBrilakis et al. titled “Methods and apparatus for processing imagestreams,” U.S. Pat. No. 8,855,406 to Lim, et al. titled “Egomotion usingassorted features,” the disclosures of which are incorporated in theirentireties by this reference. Notably, the methodologies disclosed ineach of these references require the use of two cameras to capture 2Dimages from which a 3D digital representation can thereby be obtained.

An example of fairly accurate passive photogrammetry that utilizesmultiple images generated from a single camera is provided byPhotomodeler (photomodeler.com). This software product allows a user togenerate a 3D digital representation of an object of interest frommultiple overlapping images, where the relevant detail is provided bythe orientation of images in a known area of space. In someimplementations, accurate measurements can be obtained from the 3Ddigital representations of the object(s) of interest. However,Photomodeler requires a user to conduct explicit calibration that occursin a separate step to achieve such accuracy. Once the 3D orientation isobtained, measurement and other detail information regarding the objectof interest can be provided for use. At least part of this calibrationstep comprises users perform a manual boundary identification. Thiscalibration process is time consuming, currently requiring the user togenerate a chessboard marker comprising a minimum number of images takenfrom different angles and distances with respect to the image-capturedevice, whereby more images will provide more accurate calibration.Moreover, to measure objects of interest that are longer distances fromthe camera, accurate measurements of the object of interest requirelarger calibration surface (e.g., about 6 ft.×about 6 ft. (1.82 metersby 1.82 meters)). As might be recognized, this physical calibration stepprovides the information necessary to orient the object(s) of interestin space so as to make it possible to provide 3D digital representationsof the object(s) of interest thereof so that measurements can beobtained.

Recently issued U.S. Pat. No. 8,953,024, the disclosure of which isincorporated herein in its entirety, indicates that 3D digital models ofscenes can be generated using a passive digital video camera using, inone implementation, structure from motion algorithms. Among otherthings, there is no disclosure in the '024 patent that sufficient detailabout individual objects present in the scene can be obtained to allowspecific parameters of such objects to be resolved in order to obtainaccurate 3D digital representations suitable to provide measurements orthe like.

In light of these and other issues, there remains a need forimprovements in photogrammetry that allow a user to obtain accurate 3Ddigital representations of an object of interest (or a collection ofobjects of interest) without the need for use of two camera imageacquisition and/or the use of cumbersome processing steps. Stillfurther, it would be desirable to have method and devices to obtainaccurate 3D digital representations of the object(s) using a singleimage-capture device, such as those integrated into mobile devices(e.g., smart phones, tablets, etc.). Yet further, it would be desirableto be able to obtain substantially accurate measurements of object(s) ofinterest in a scene. The present invention provides this and otherbenefits.

SUMMARY OF THE INVENTION

In one embodiment, the invention provides a method for generating 3Ddigital representations of an object of interest using an image-capturedevice. An exemplary method comprises receiving a plurality of 2Ddigital images of a scene, where at least one object of interest ispresent in the scene. The 2D digital images will at least partiallyoverlap with regard to the object of interest. In order to generate the3D digital representation of the object of interest, at least some ofthe 2D digital overlapping images of the object are processed usingmethodology that incorporates a structure from motion algorithm. The 3Ddigital representations obtained in accordance with the invention aresuitable for generating one or more of a 3D model, a 3D point cloud, a3D line cloud or a 3D edge cloud, wherein each, independently,corresponds to one or more dimensions in the object. Further, the 3Ddigital representation, and any data or other information obtainabletherefrom, is accurate in relation to the dimensions of the actualobject, which allows substantially accurate measurements of one or moredimensions of the object to be obtained.

In a further embodiment, the invention provides a method of detectingboundaries in at least one object of interest in a scene. In thisregard, overlapping 2D digital images of an object of interest in ascene are generated. Boundary detection information regarding the objectis generated from a process that incorporates a structure from motionalgorithm. With respect to the object of interest in the scene, theboundary detection information can be used provide measurements, 3Ddigital representations, 3D point clouds, 3D line clouds, 3D edge cloudsand the like.

The overlapping 2D digital images used in the present invention can beobtained from a single image-capture device. Still further, the singleimage-capture device is a video camera. The 2D digital images can begenerated by an image-capture device that comprises a passive sensingtechnique. Yet further, the 2D digital images can be generated by animage-capture device that consists essentially of a passive sensingtechnique. The image-capture devices can be integrated into a devicesuch as a smartphone, tablet or wearable device or the image-capturedevices can be as stand-alone camera device. The image-capture devicecan also be incorporated in a specialized measurement device.Accordingly, the present invention relates to one or more devices thatincorporate the methods herein.

Additional advantages of the invention will be set forth in part in thedescription that follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. Theadvantages of the invention will be realized and attained by means ofthe elements and combination particularly pointed out in the appendedclaims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a system 101 according to some embodimentsof the present invention.

FIG. 1B is a flowchart of a method 102 illustrating an exemplary methodto obtain 3D digital representations of an object of interest accordingto the methodology herein.

FIG. 2 is a flowchart of a method 201 illustrating an exemplary methodto perform the structure-recovery portion 125 of the process of FIG. 1.

FIG. 3 is a flowchart of a method 301 illustrating an exemplarymethodology for use in navigation applications for robots and the like.

FIG. 4 is a flowchart of a method 401 illustrating an exemplary methodto perform a simultaneous-localization-and-mapping (SLAM) portion 325 ofmethod 301 of FIG. 3.

DETAILED DESCRIPTION OF THE INVENTION

Many aspects of the disclosure can be better understood with referenceto the Figures presented herewith. The Figures are intended toillustrate the various features of the present disclosure. Moreover,like references in the drawings designate corresponding parts among theseveral views. While several implementations may be described inconnection with the included drawings, there is no intent to limit thedisclosure to the implementations disclosed herein. To the contrary, theintent is to cover all alternatives, modifications, and equivalents.

The term “substantially” is meant to permit deviations from thedescriptive term that do not negatively impact the intended purpose. Alldescriptive terms used herein are implicitly understood to be modifiedby the word “substantially,” even if the descriptive term is notexplicitly modified by the word “substantially.”

In one embodiment, the invention provides a method for generating 3Ddigital representations of an object of interest in a scene from animage-capture device. An exemplary method comprises receiving aplurality of 2D digital images of the scene, where at least one objectof interest is present in the scene. The 2D digital images will at leastpartially overlap with regard to the object of interest. In order togenerate the 3D digital representation of the object of interest, atleast some of the 2D digital overlapping images of the object areprocessed using methodology that incorporates a structure from motionalgorithm. The 3D digital representations obtained in accordance withthe invention are suitable for generating one or more of a 3D model, a3D point cloud, a 3D line cloud or a 3D edge cloud, wherein each,independently, corresponds to one or more dimensions in the object.Further, the 3D digital representations, and any data or otherinformation obtainable therefrom, is accurate in relation to thedimensions of the actual object, which allows accurate measurements ofone or more dimensions of the object to be obtained.

As used herein, “overlapping images” means individual images that each,independently, include at least one object of interest, where suchimages overlap each other as to one or more dimensions of the object ofinterest are concerned. “Overlapping” in relation to the inventionherein is described in further detail hereinbelow.

As used herein, an “object of interest” encompasses a wide variety ofobjects such as, for example, structures, parts of structures,landscapes, vehicles, people, animals and the like. Indeed, “object ofinterest” can be anything from which a 2D image can be obtained and thatfrom which information suitable for generation of accurate 3D digitalrepresentations of such objects can be obtained according to themethodology herein. The at least one object of interest can havemultiple dimensions, such as linear or spatial dimensions, some or allof which may be of interest, such as to provide measurements or otheruseful information. Further, the methodology herein can be utilized togenerate accurate 3D digital representations of more than one object ofinterest in a scene, such as a collection of smaller objects (e.g.,doors, windows, etc.) associated with a larger object (e.g., the overalldimensions of a building) where such collection of smaller and largerobjects are present in the plurality of overlapping 2D images in ascene.

The at least one object of interest, for example, can be a roof on astructure that is present in a scene that includes the structure,landscaping and other objects. The length of the roof on a front side ofthe structure (such as in meters or feet, etc.) could be at least onedimension of interest. Alternatively, each of the dimensions of the roof(such as length on the back, front and sides of the structure and thepitch) could comprise a plurality of dimensions of interest. As would berecognized, these one or plurality of dimensions/features will have anactual measurement value that will be obtainable when a physicalmeasurement of the length, depth, etc., is conducted, such as by alinear measurement tool or an electronic distance measurement tool.

The overlapping 2D digital images used in the present invention can beobtained from a single image-capture device. Still further, the singleimage-capture device is a video camera. The 2D digital images can begenerated by an image-capture device that comprises a passive sensingtechnique. Yet further, the 2D digital images can be generated by animage-capture device that consists essentially of a passive sensingtechnique. The image-capture devices can be integrated into a devicesuch as a smartphone, tablet or wearable device or the image-capturedevices can be as stand-alone camera device. The image-capture devicecan also be incorporated in a specialized measurement device.Accordingly, the present invention relates to one or more devices thatincorporate the methods herein.

In accordance with the methods herein, an extracted measurement value ofthe one or a plurality of dimensions in the object of interest and otheruseful information, such as boundary detection information as discussedherein below, can be obtained from using a single passive image-capturedevice, such as that integrated into a smartphone, tablet, wearabledevice, digital camera (for example, digital cameras on drones) or thelike.

When the plurality of overlapping 2D images are derived from avideo-image-capture device, the images will be overlapping. As usedherein, “video” means generally that the images are taken, for example,as single frames in quick succession for playback to provide theillusion of motion to a viewer. In some aspects, video suitable for usein the present invention comprises at least about 24 frames per second(“fps”), or at least about 28 fps or at least about 30 fps (frames persecond) or any suitable fps as appropriate in a specific context.

As used herein, “image-capture-device calibration” is the process ofdetermining internal image-capture-device parameters (e.g., focallength, skew, principal point, and lens distortion) from a plurality ofimages taken of an object with known dimensions (e.g., a planar surfacewith a chessboard pattern). Image-capture-device calibration is used forrelating image-capture device measurements with measurements in the real“3D” world. Objects in the real world are not only three-dimensional,they are also physical spaces with physical units. Hence, the relationbetween the image-capture device's natural units (pixels) and the unitsof the physical world (e.g., meters) can be a significant component inany attempt to reconstruct a 3D scene and/or an object incorporatedtherein. A “calibrated image-capture device” is an image-capture devicethat has undergone a calibration process. Similarly, an “uncalibratedimage-capture device” is an image-capture device that has not been putthrough a calibration process, in that no information or substantiallyno information regarding the internal image-capture device parameters isprovided and substantially the only available information about theimages is presented in the image/video frame itself. In someembodiments, the present invention incorporates a calibratedimage-capture device. In other embodiments, the present inventionincorporates an uncalibrated image-capture device. In some embodiments,the present invention extracts metadata (such as EXIF tags) thatincludes camera-lens data, focal length data, time data, and/or GPSdata, and uses that additional data to further process the images intopoint-edge-cloud data.

In accordance with some aspects of the invention herein, use of aplurality of 2D overlapping images derived from video greatly improvesthe ease and quality of user capture of the plurality of 2D images thatcan be processed to provide accurate 3D digital representations of theat least one object of interest, for example, such as to generatesubstantially accurate measurements of the object. As one example ofthis improvement, the sequential nature of video has been found by theinventors herein to improve 3D digital representation quality due to anattendant reduction in the errors associated with a user needing toobtain proper overlap of the plurality of overlapping 2D images so thatdetailed information about the object of interest can be derived.Another advantage of the present invention is the shortened time neededto obtain the overlapping 2D images used in the present invention tocreate detailed information about the object of interest such that anaccurate 3D digital representation can be obtained for use. Stillfurther, the inventors herein have found that use of video as the sourceof the plurality of overlapping 2D images can allow tracking of pointsthat are inside (i.e., tracking points within the boundaries of theimages) or outside of the images of the object of interest (i.e.,continuing to track points that are first “followed” when in the imageframe, and then tracking estimated positions of those points no longerin the images intermediate in time (the points have moved outside theboundaries of the images), so that when those points are in the field ofview of later image frames, the later-followed points can besubstantially correlated to those same features in the earlier imageframes), where such point tracking provides improvements in the 2D-imagedata used to generate the 3D digital representations the at least oneobject of interest in a scene. In turn, it has been found that thequality of the 3D digital representations of the object(s) of interestherein can be improved.

While the present invention is particularly suitable for use withimage-capture devices that generate a video from which overlapping 2Dimages can be provided, the present invention is not limited to the useof video. That is, the plurality of overlapping 2D images can suitablybe provided by an image-capture device that provides 2D still images,such as a “point and shoot” digital camera. When using a such a digitalstill camera, the at least two overlapping images can be obtained fromimages that comprise a suitable parallax between and amongst the imagesto allow generation of information from which an accurate 3D digitalrepresentations of the object(s) can be obtained.

As would be recognized, a plurality of still 2D images taken in sequencecan also be defined as “video” if played back at a speed that allows theperception of motion. Therefore, in some aspects, the plurality ofoverlapping 2D images can be derived from a plurality of digital stillimages and/or from video without affecting the substance of the presentinvention, as long as the plurality of overlapping 2D images thatinclude an object of interest can be suitably processed to generatedetailed information from which the accurate 3D digital representationsof the object(s) of interest can be generated.

The overlapping 2D images of a scene will include at least a portion ofthe at least one object of interest. In accordance with the invention,at least a portion of the overlapping 2D images of the scene will alsobe overlapping with regard to the at least one object of interest.

In some aspects, the plurality of overlapping 2D images includes atleast two (2) suitably overlapping 2D images, where the overlap is inrelation to the at least one object of interest. In other embodiments,the plurality of overlapping 2D images includes at least 5, at least 10,or at least 15 or at least 20 suitably overlapping 2D images, where theoverlap is in relation to the at least one object of interest. As wouldbe recognized, the number of overlapping 2D images needed to generate anaccurate 3D digital representations of the object(s) of interest in ascene will depend, in part, on factors such as the size, texture,illumination and potential occlusions of the object of interest, as wellas the distance of the object of interest from the image-capture device.

As noted, sequential images extracted from video will possess overlap.The overlap present in sequential images generated from video willdepend, in part, on the speed at which the user moves the image-capturedevice around the at least one object of interest and the orientation ofthe image-capture device in space with reference to the object ofinterest.

When the image-capture device is a still digital camera, the 2D imagescan be made suitably overlapping with regard to the at least one objectof interest using one or more methods known to one of ordinary skill inthe art, such as, in some embodiments, the camera operator taking thesuccessive still images including the at least one object of interestwhile changing the angular orientation, the linear location, thedistance, or a combination thereof in a manner that has the object ofinterest in each successive image captured. In this regard, theplurality of overlapping 2D images are suitably processable to allowaccurate 3D digital representations of the at least one object ofinterest to be derived therefrom.

To provide suitably overlapping 2D images incorporating the at least oneobject of interest from sources other than video, the individual imagescan be overlapped, where such overlap is, in reference to the at leastone object of interest, at least about 50% or at least about 60% or atleast about 70% or at least about 80% or at least about 90%. In someembodiments, the amount of overlap in the individual images in theplurality of overlapping 2D images, as well as the total number ofimages needed to provide an accurate digital representation of theobject of interest, will also depend, in part, on the relevant featuresof the object(s). In some embodiments, such relevant features include,for example, the amount of randomness in the object shape, the textureof and size of the at least one object of interest relative to theimage-capture device, as well as the complexity and other features ofthe overall scene.

In a further aspect, the present invention comprises image-capturedevices comprising passive sensing techniques and methods relatingthereto utilizing a plurality of overlapping 2D images suitable forgenerating accurate 3D digital representations the at least one objectof interest in a scene. The inventors herein have found that accurate 3Ddigital representations of the object(s) present in a scene can beobtained using a plurality of overlapping 2D images incorporating theobject(s) substantially without the use of an active sensor/signalsource, such as a laser scanner or the like. As would be understood byone of ordinary skill in the art, “passive-image-capture devices” meansthat substantially no active signal source such as a laser or structuredlight (as opposed to camera flash or general-illumination devices) orsound or other reflective or responsive signal is utilized to measure orotherwise sense at least one object of interest so as to provide theinformation needed to generate the accurate 3D digital representationsof the at least one object of interest present in a scene.

As used herein, “accurate” in relation to the 3D digital representationsof the at least one object of interest comprises, in part, data or otherinformation from which substantially accurate measurements of theobject(s) can be obtained as defined elsewhere herein.

In some embodiments, the present invention further includes passivephotogrammetry techniques where the images are obtained from a singleimage-capture device. Yet further, in some embodiments, no more than onepassive image-capture device is used in accordance with the methodsherein. This use of images from only a single image-capture device is incontrast to the traditional use of at least two cameras or one or moreprojectors used to obtain 3D digital representations of the at least oneobject of interest passive sensing methods as disclosed, for example, inUS Patent Publication No. US2013/0083990 and PCT Publication No.WO2013/173383, which are each incorporated by reference as set forthpreviously. In particular, and as would be recognized by one of ordinaryskill in the art, prior-art passive image-capture devices used togenerate 3D digital representations of the at least one object ofinterest utilize at least two cameras (or projectors) displaced in adirection away from one another (e.g., horizontally) so as to obtain atleast two differing views of a scene and any objects included therein.By comparing these at-least-two images obtained from two image-capturedevices, the relative depth information of the scene and/or objectspresent therein can be obtained for display to a viewer with or withoutprocessing of the image there between.

Yet further, prior art methods perform poorly if the motion between twoframes is too small or limited. In contrast, the methodology hereinleverages such small or limited motions and creates improved resultssuch situations.

In further embodiments, the present invention includes methods of usingmobile devices configured with passive image-acquisition capabilitysuitable to provide accurate 3D digital representations of the at leastone object of interest in a scene. In some embodiments, the methodologyherein can be utilized to provide measurements and other usefulinformation regarding the object(s). Yet further, the present inventionincludes methods of using mobile devices configured with passiveimage-acquisition technology, whereby substantially accuratemeasurements of one or more dimensions of the objects can be obtained.

In some embodiments, point clouds that incorporate information regardingthe at least one object of interest are generated using conventionalmethods. As used herein, a “point cloud” is a set of data points in thesame coordinate system. In a three-dimensional coordinate system, thesepoints are usually defined by X, Y, and Z coordinates. In otherembodiments, the inventors herein have found that inventive point cloudscan be obtained, where such inventive point clouds further includeadditional data representative of edge information in the object ofinterest. Yet further, one or more of point clouds, edge clouds and lineclouds are obtainable according to the methodology herein, wherein eachof these aspects can include data or other information from whichmeasurements or other useful information about the at least one objectof interest can be generated. “Edge cloud” is a set of edge points inthe same coordinate system and represented by X, Y, and Z and compriseone or more discontinuities in depth, surface, orientation, reflection,or illumination. “Line cloud” is a set of 3D straight lines in the samecoordinate system. Each line can be defined using its two end points orPlucker coordinates.

Unlike prior art methodologies that utilize point-level data to generate3D digital representations, in some aspects, the present invention can,in some circumstances, be characterized as “hybrid” in nature in that itis possible to utilize any combination of points, edges, and lines(point+edge+line, point+edge, point+line, edge+line, etc.). As a result,while the prior art can only produce point clouds, with the inventionherein it is possible to create point clouds, line clouds, and edgeclouds and any combination thereof. Moreover, the prior art solutionsonly produce a point cloud with an unknown scale. Therefore, 3Dmeasurements cannot be extracted directly from the point cloud. Thepoint cloud has to be scaled first. In the present invention, 3Dmeasurements can, in some embodiments, be extracted directly from thepoint cloud, line cloud and/or edge cloud data substantially without theneed for a scaling step.

In another aspect, the accurate 3D digital representations of the atleast one object of interest are generated by processing overlapping2D-image data generated from one or more discontinuities in depth,surface, orientation, reflection, or illumination, wherein such imagedata is derived from a plurality of overlapping 2D images of an objectof interest.

In one aspect, a suitable methodology (albeit where two image-capturedevices are required to provide dimensions of the object of interest)that can be used for structure recovery is described in US PatentPublication No. 2013/0083990, previously incorporated by reference.

The methodology herein, in some aspects, utilizes data or otherinformation extracted from a plurality of overlapping 2D images tocreate a robust data set for image processing wherein a plurality oflines, edges and points included therein are specific to lines, edgesand points corresponding to the at least one object of interest asincorporated in the plurality of 2D overlapping images of the object. Insome contexts, it has been found that the inventive methodology canprovide one or more of the following improvements over the prior art: 1)the edge-detection method substantially filters out useless data, noise,and frequencies while preserving the important structural properties ofthe at least one object of interest; 2) the amount of data needed toprovide an accurate 3D digital representation of an object is reduced,as is the need for attendant data processing; and 3) the necessaryinformation needed for object detection and segmentation (i.e., objectboundaries) is provided which is an unmet need in Building InformationModeling (BIM). As used herein, “BIM” means an object-orientedbuilding-development tool that utilizes modeling concepts, informationtechnology and software interoperability to design, construct andoperate a building project, as well as communicate its details. Furtherimprovements are found in the present invention are found from thesubstantially simultaneous processing of point, line and edge data.Prior art solutions only produce a dense 3D point cloud at thebeginning, and in a successive step they extract edge points from thegenerated 3D dense point cloud.

In some embodiments, the 2D digital images suitable for use in thepresent invention may be missing some or all of the information storedin EXIF tags. This can allow images other than JPEG images to be used asinput data in the present invention.

In a further embodiment, the invention provides a method of detectingboundaries in an object of interest in a scene. In this regard,overlapping 2D digital images of an object in a scene are generated.Boundary detection information regarding the object is generated from aprocess that incorporates a structure from motion algorithm. Withrespect to the at least one object of interest in the scene, theboundary detection information can be used to generate measurements, 3Ddigital representations, 3D point clouds, 3D line clouds, 3D edge cloudsand the like.

A “boundary” is a contour in the image plane that represents a change inpixel ownership from one object surface to another. “Boundary pixels”mark the transition from one relatively constant region to another,where the constant region can comprise one or more of an object ofinterest or a scene in which the object appears in the image. Boundarydetection is a computer vision problem with broad applicability in areassuch as feature extraction, contour grouping, symmetry detection,segmentation of image regions, object recognition, categorization andthe like. Detecting boundaries is significantly different from simpleedge detection, where “edge detection” is a low level technique todetect an abrupt change in some image feature, such as brightness orcolor. In contrast, boundary detection relates to the detection of moreglobal properties, such as texture and, therefore, involves integrationof information across an image. So, for example, a heavily texturedregion might give rise to many edges, but to suitably provideinformation suitable to generate a 3D digital representation of anobject of interest therefrom, there should be substantially no boundarydefined within the textured region. Moreover, accurate boundarydetection is needed to resolve discontinuities in depth that allowaccurate rendering of 3D digital representations.

In some embodiments, information needed to generate accurate 3D digitalrepresentations of the at least one object of interest in a scene can bedetermined using a “structure from motion” algorithm. As would berecognized, a structure from motion algorithm can be used to extract 3Dgeometry information from a plurality of overlapping images of an objector a scene. In accordance with the present invention, information neededto provide accurate 3D digital representations of the object(s) ofinterest of the object can be generated from a process that incorporatesa structure from motion algorithm that estimates camera positions foreach image frame in the plurality of overlapping images. As would berecognized, many structure from motion algorithms incorporate key-pointdetection and matching, so as to form consistent matching tracks andallowing the solving for camera parameters.

The inventors herein have developed improvements in the ability toanalyze the plurality of 2D overlapping images of at least one object ofinterest, where such improvements, in some embodiments, assist in thegeneration of 3D digital representations of the object. In this regard,an inventive methodology comprises parameterizing a line with twoend-points. This parameterization step provides two advantages overexisting line- or point-based 3D reconstruction methodologies, such asthose provided by prior art structure from motion algorithms, becausethe inventive methods are able to achieve the following.

First, a duality is created between points and lines that is preservedby: a) visual triangulation for calculating 3D coordinates of features(point and lines) and b) reprojecting 3D features into the 2D-imageplane. This duality allows interchanging the role of points and linesthe mathematical formulations whenever appropriate.

Second, a parameterization step facilitates modeling of lens-distortionparameters even when substantially only line level information ispresent. Due to deviations from rectilinear projection caused by lensdistortion, straight lines in a scene are typically transformed intocurves in the image of the scene. Existing line-based 3D-reconstructionalgorithms assume that the input data (images or video frames) arealready undistorted; this necessitates use of pre-calibrated cameras inprior-art methods. In some embodiments, substantially no such assumptionis made in the present invention. As such, uncalibrated cameras areparticularly suitable for use in the present invention.

Further in regards to the parameterization, the present invention allowsreprojection errors to be calculated with a weighing function thatsubstantially does not over- or underestimate the contribution of lineand/or edge points to the total reprojection error cost function.

In some embodiments, if

l _(s) ^(d)=(u _(s) ^(d) ,v _(s) ^(d))^(T) and l _(e) ^(d)=(u _(e) ^(d),v _(e) ^(d))^(T)

denote the distorted 2D coordinates of two end points of a line segmentin an image,

L _(s)=(X _(s) ,Y _(s) ,Z _(s))^(T) and L _(e)=(X _(e) ,Y _(e) ,Z_(e))^(T)

denote their corresponding 3D points and P^(3×4) is thecamera-projection matrix.In some embodiments, an information processor that uses the presentinvention performs the following:

-   1) Calculates

l _(s) ^(u)=(u _(s) ^(u) ,v _(s) ^(u))^(T) and l _(e) ^(u)=(u _(e) ^(u),v _(e) ^(u))

-   -   which are the undistorted coordinates of the two end points.

-   2) Locates a 3D point on the infinite 3D line that is connecting the    two end points (P_(s)) using l_(s) ^(u), L_(s), L_(e), and P^(3×4)

$\mspace{20mu} {a_{1} = {\begin{bmatrix}{P^{00} - {P^{20} \times u_{s}^{u}}} & {P^{01} - {P^{21} \times u_{s}^{u}}} & {P^{02} - {P^{22} \times u_{s}^{u}}}\end{bmatrix}\begin{bmatrix}{X_{e} - X_{s}} \\{Y_{e} - Y_{s}} \\{Z_{e} - Z_{s}}\end{bmatrix}}}$ $\mspace{20mu} {a_{2} = {\begin{bmatrix}{P^{10} - {P^{20} \times v_{s}^{u}}} & {P^{11} - {P^{21} \times v_{s}^{u}}} & {P^{12} - {P^{22} \times v_{s}^{u}}}\end{bmatrix}\begin{bmatrix}{X_{e} - X_{s}} \\{Y_{e} - Y_{s}} \\{Z_{e} - Z_{s}}\end{bmatrix}}}$ $b_{1} = {\begin{bmatrix}{{P^{20} \times u_{s}^{u}} - P^{00}} & {{P^{21} \times u_{s}^{u}} - P^{01}} & {{P^{22} \times u_{s}^{u}} - P^{02}} & {{P^{23} \times u_{s}^{u}} - P^{03}}\end{bmatrix}{\quad{{\begin{bmatrix}X_{s} \\Y_{s} \\Z_{s} \\1\end{bmatrix}b_{2}} = {\begin{bmatrix}{{P^{20} \times v_{s}^{u}} - P^{10}} & {{P^{21} \times v_{s}^{u}} - P^{11}} & {{P^{22} \times v_{s}^{u}} - P^{12}} & {{P^{23} \times v_{s}^{u}} - P^{13}}\end{bmatrix}{\quad{{\begin{bmatrix}X_{s} \\Y_{s} \\Z_{s} \\1\end{bmatrix}\mspace{20mu} t} = {{\frac{{a_{1} \times b_{1}} + {a_{2} \times b_{2}}}{{a_{1} \times a_{1}} + {a_{2} \times a_{2}}}\mspace{20mu} P_{s}} = \begin{bmatrix}{X_{s} + {\left( {X_{e} - X_{s}} \right) \times t}} \\{Y_{s} + {\left( {Y_{e} - Y_{s}} \right) \times t}} \\{Z_{s} + {\left( {Z_{e} - Z_{s}} \right) \times t}}\end{bmatrix}}}}}}}}$

-   3) Locates a 3D point on the infinite 3D line that is connecting the    two end points (P_(e)) using l_(e) ^(u), L_(s), L_(e), and P^(3×4)    -   following a similar process presented in the previous step.-   4) Projects P_(s) and P_(e) into the 2D-image plane using P^(3×4) to    get p_(s) and p_(e).-   5) Finds l_(h) which is a normalized homogeneous line that connects    p_(s) and p_(e).-   6) Calculates the reprojection error.

e=0.5×(|

l _(n) ,p _(s)

|+|

l _(n) ,p _(e)

|)

-   -   where ∥ represents the absolute value and        represents the inner product.

FIG. 1A is a block diagram of a system 101 according to some embodimentsof the present invention. In some embodiments, system 101 includes oneor more cloud-computing servers 181 each connected to the internet 180.In some embodiments, a non-transitory computer-readable storage medium183 has photogrammetry instructions and data structures of the presentinvention stored thereon. In some embodiments, the methods of thepresent invention execute on cloud-computing server(s) 181 using thephotogrammetry instructions and data structures from computer-readablestorage medium 183, wherein a user 98 uploads images from still camera182 and/or video camera 184 into cloud-computing server(s) 181, eitherdirectly (e.g., using the cell-phone or other wireless network, orthrough a conventional personal computer 186 connected to the internet).In other embodiments, photogrammetry instructions and data structures ofthe present invention are transmitted from computer-readable storagemedium 183 into local non-transitory computer-readable storage media 187(such as rotating optical media (e.g., CDROMs or DVDs) or solid-statememory devices (such as SDHC (secure data high-capacity) FLASH devices),which are connected to, plugged into, and/or built into cameras 182 or184 or conventional personal computers 186 to convert such devices fromgeneric information processors into special-purpose systems that convertimage data into photogrammetry data according to the present invention.In some embodiments, system 101 omits one or more of the devices shownand still executes the methods of the present invention.

FIG. 1B presents a flowchart of a method 102 illustrating one aspect ofthe present invention. In FIG. 1B, FIG. 2, FIG. 3, and FIG. 4,rectangular boxes represent a function and ovals are the inputs/outputs.In block 100 a plurality of overlapping images are received by. Theseimages can be derived from a still image-capture device 182 or a videoimage-capture device 184 as discussed elsewhere herein. In someembodiments, feature lines are detected and matched/tracked in block 105and block 110, respectively. The output of the detection and matchingprocess of block 110 and block 105 are corresponding lines of block 115and corresponding points of block 120. In some embodiments, in block125, methods such as linear methods, are used for structure recoveryprocesses, such as those presented in more detail in reference to FIG.2. In some embodiments, an initial estimation of structure and motiondata in block 130 is determined based on the structure recovery of block125. In some embodiments, hybrid bundle adjustment techniques in block135 are used to further refine/optimize the 3D structure and motion data140 and are used in process 145 to generate a 3D point, line and/or edgecloud 150 representative of the at least one object of interest. In someembodiments, 3D structure and motion data 140 are used in a 3D planedetection process 155 to detect 3D planes 160. In some embodiments, the3D point, line and/or edge cloud of block 150 and 3D plane of block 160are included in intelligent data smoothing in block 165 to generate a 3Ddigital representation of block 170 incorporating the at least oneobject of interest.

FIG. 2 is a flowchart of a method 201 illustrating an exemplary methodto perform the structure-recovery portion 125 of the process of FIG. 1B.In comparison to previous methodologies using passive image-capturedevices, in some aspects, the present invention provides notablebenefits relating to the ability to utilize a single image capturedevice to generate the plurality of overlapping images. In this regard,FIG. 2 illustrates such benefits in relation to structure recovery ofblock 125 called out from FIG. 1. In block 200 pairwise epipolargeometries are computed, and used in block 205 to build a graph ofepipolar geometries. In some embodiments, from the data from which thegraph of block 205 is created, the confidence level for each epipolargeometry is calculated in 210. Where the calculated epipolar geometriesare determined to meet the desired confidence level (such as 90% or 95%or 99% confidence level), a connectivity graph is built in block 215. Inblock 220, the relative rotations of the various points on theconnectivity graph of block 215 are estimated, followed by calculationof global rotations in block 225. In block 230, the relative translationand scaling factor for the resulting data is determined, whereby thedata generated in method 201 of FIG. 2 is used to provide an initialestimation of structure and motion 130 for further application to theprocess set out in FIG. 1.

Further in regard to block 225 wherein the initial estimate of therelative rotation for each pair of images or video frames (those whichresulted in confident epipolar geometries) are estimated, in someembodiments, the global rotation for each view is calculated by thefollowing methodology:

1) Build matrix A (wherein A is a (3m)×(3m) matrix and m is the totalnumber of images or video frames)

If the relative rotation between view i and j is denoted by R_(ij), then

for int i=0; i<m−1; i++ { for int j=i+1; j<m; j++ {  ${A\text{+}} = \begin{bmatrix}\ldots & . & . & . & \ldots & . & . & . & \ldots \\\ldots & 1 & 0 & 0 & \ldots & {- R_{ij}^{00}} & {- R_{ij}^{01}} & {- R_{ij}^{02}} & \ldots \\\ldots & 0 & 1 & 0 & \ldots & {- R_{ij}^{10}} & {- R_{ij}^{11}} & {- R_{ij}^{12}} & \ldots \\\ldots & 0 & 0 & 1 & \ldots & {- R_{ij}^{20}} & {- R_{ij}^{21}} & {- R_{ij}^{22}} & \ldots \\\ldots & . & . & . & \ldots & . & . & . & \ldots\end{bmatrix}$  as an example to clarify matrix indices: A_(3i,3i) = 1and A_(3j,3j) = −R_(ij) ⁰⁰  } }

2) Singular Value Decomposition (SVD) of A

A=UΣV*

With regard to block 230 where the global rotation is calculated, thefollowing methodology is used.1) Calculate global rotation for view i from V* matrix

$V^{*} = \begin{bmatrix}\ldots & \ldots & \ldots & \ldots & \ldots \\\ldots & v_{{{3\; i} + 0},{{3\; m} - 3}} & v_{{{3\; i} + 1},{{3\; m} - 3}} & v_{{{3\; i} + 2},{{3\; m} - 3}} & \ldots \\\ldots & v_{{{3\; i} + 0},{{3\; m} - 2}} & v_{{{3\; i} + 1},{{3\; m} - 2}} & v_{{{3\; i} + 2},{{3\; m} - 2}} & \ldots \\\ldots & v_{{{3\; i} + 0},{{3\; m} - 1}} & v_{{{3\; i} + 1},{{3\; m} - 1}} & v_{{{3\; i} + 2},{{3\; m} - 1}} & \ldots\end{bmatrix}$ $R_{i} = \left\{ \begin{matrix}v_{{{3\; i} + 0},{{3\; m} - 1}} & v_{{{3\; i} + 1},{{3\; m} - 1}} & v_{{{3\; i} + 2},{{3\; m} - 1}} \\v_{{{3\; i} + 0},{{3\; m} - 2}} & v_{{{3\; i} + 1},{{3\; m} - 2}} & v_{{{3\; i} + 2},{{3\; m} - 2}} \\v_{{{3\; i} + 0},{{3\; m} - 3}} & v_{{{3\; i} + 1},{{3\; m} - 3}} & v_{{{3\; i} + 2},{{3\; m} - 3}}\end{matrix} \right\rbrack$

2) Find the closest orthogonal matrix to R_(i).

Once the scale ambiguity of, for example, a 3D point cloud is resolved,the spatial distance between each point pair in the point cloud willrepresent the distance between the corresponding physical points in theactual scene. In some embodiments, this is leveraged to extract a widevariety of dimensions and measurements from the point cloud. In someembodiments, the obtained knowledge about corner points, edge/boundarypoints, blobs, ridges, straight lines, curved boundaries, planarsurfaces, curved surfaces, and other primitive geometry elements canprovide the capability to identify significant parts of the scene andautomatically extract corresponding measurements (length, area, volume,etc.). In some embodiments, the 2D locations of these primitivegeometries are first detected in images or video frames. The image-basedcoordinates are then converted into 3D coordinates via the calculatedcamera matrices.

As mentioned, the present invention provides accurate 3D digitalrepresentations of at least one object in a scene. In one aspect, thelevel of accuracy of the 3D digital representations of the object(s) ofinterest is with reference to one or more of the actual dimensions ofthe object of interest. In this regard, at least one object of interestis identified, selected or otherwise specified, where theidentification, etc., can include identification of at least onedimension of interest in the object, or such identification, etc., mayinclude a plurality of dimensions of interest where each of thesedimensions, independently, includes an actual value. As discussedelsewhere herein, the identification, etc., of the at least one objectof interest and/or the one or more dimensions in the object(s) can be byeither or both of a computer or a user.

In some embodiments, the accuracy of the measurements obtained accordingto the invention herein can be characterized in relation to a specifiednumber of pixels. The methodology herein allows a user to obtainmeasurements of one or more dimensions of the object of interest of upto and including a 1.0 pixel standard deviation or, in otherembodiments, a 0.5 pixel standard deviation is provided. As would berecognized, pixel size is an aspect of the image-capture devicespecifications and the distance of the image-capture device from theobject of interest. This is illustrated in Table 2 hereinbelow.

In some embodiments, accuracy in pixels relative to the actualdimensions of the object of interest is represented according to thefollowing formula:

Pixel size in object=(distance of object of interest from IC device)*(ICdevice sensor size)/((IC device resolution*IC device focal length))

The IC (“image capture”) device sensor size, resolution and focal lengthare features or characteristics of each image-capture device. Forexample, the below Table 1 sets out some representative specificationsfor existing image-capture devices:

TABLE 1 Image-Capture Device Specifications focal Length (mm) sensor(mm) Resolution (pixels) iPhone 5s (Apple) 4.12 4.80 1920 GoPro H3(GoPro) 21 (Medium 6.17 1920 Field of View)

The calculations in Table 2 (and Table 3) assume that there is noscaling error. As would be recognized, scaling error will decrease theaccuracy of the measurement derived from the image-capture device vs.the actual measurement of the one or more dimensions of the object ofinterest.

TABLE 2 US Metric Pixel size 0.5 Std dev in 0.5 Std dev in PixelDistance object Distance object size in Pixel size in object tomeasurements object to measurements image at image at camera (ft) (in)camera (m) (cm) distance (mm) distance (in) iPhone 5s 15 0.05 4.6 0.142.77 0.11 30 0.11 9.1 0.28 5.55 0.22 60 0.22 18.3 0.55 11.10 0.44 1200.44 36.6 1.11 22.19 0.87 240 0.87 73.1 2.22 44.39 1.75 480 1.75 146.34.44 88.77 3.49 960 3.49 292.6 8.88 177.54 6.99 1920 6.99 585.2 17.75355.09 13.98 GoPro H3 15 0.01 4.6 0.03 0.70 0.03 30 0.03 9.1 0.07 1.400.06 60 0.06 18.3 0.14 2.80 0.11 120 0.11 36.6 0.28 5.60 0.22 240 0.2273.1 0.56 11.19 0.44 480 0.44 146.3 1.12 22.39 0.88 960 0.88 292.6 2.2444.77 1.76 1920 1.76 585.2 4.48 89.55 3.53

It should also be noted that the stated pixel errors are in relation tothe general capabilities of image-capture devices available in themarket currently. In the future, image-capture devices will be availablewith higher resolutions that will allow attendant improvements in theaccuracy of the inventive methods. Such higher-resolution devices willprovide sharper (e.g., less blurred) and less noisy images which will,in turn, result in less pixelization effect and hence the pixel noisewill be decreased to allow smaller pixel errors. Still further,improvements in image-capture devices will result in reduced lensdistortion and improvements in the squareness of pixels (i.e., withoutskews). Improvements in image sensors, as well as other relevantsensors, will also be obtained. Prospective improvements include, butare not limited to, higher focal length and smaller sensor size. Anysuch improvements individually or in combination will result inattendant improvements in the data available from 2D images and,therefore more accurate measurements will be obtainable using themethodologies herein. Such improved image-capture devices and theresulting data therefrom are contemplated for use with the inventivemethods.

In some embodiments, accuracy of the measurements derived from theimage-capture device is also represented in percent error. Themethodology herein enables measurements to be derived from theimage-capture device having accuracy within, in some embodiments, about5% or in other embodiments, about 10% or in still other embodiments,about 20% error relative to the actual measurement value of the objectof interest. In some embodiments, this error is calculated from thefollowing formula:

Error %=(((distance of object from IC device)/2)*object size)*((ICdevice sensor size)/(IC resolution*IC device focal length)*100%)

Representative % Error Calculations are Presented in Table 3

TABLE 3 US Metric Accuracy Maximum Maximum “Fabrication “EstimationRatio of Size of pixel offset Size of pixel offset Level” Level”“Distance Distance Smallest for object Distance Smallest for objectMaximum Maximum object to Camera Info object to Object this size objectto Object this size Pixel Pixel Camera”/ Focal Sensor Reso- camera with<5.0% and 5.0% camera with <5.0% and 5.0% Offset Offset “Size of Lengthsize lution (ft) err (in) err (in) (m) err (cm) err (cm) (5% error) (20%error) Smallest (mm) (mm) (pixels) iPhone 5s 15 1.1 0.05 4.6 0.3 0.010.05″ 0.22″ 165 4.12 4.80 1920 30 2.2 0.11 9.1 0.6 0.03 0.11″ 0.44″ 1654.12 4.80 1920 60 4.4 0.22 18.3 1.1 0.06 0.22″ 0.87″ 165 4.12 4.80 1920120 8.7 0.44 36.6 2.2 0.11 0.44″ 1.75″ 165 4.12 4.80 1920 240 17.5 0.8773.1 4.4 0.22 0.87″ 3.50″ 165 4.12 4.80 1920 480 35.0 1.75 146.3 8.90.44 1.75″ 6.99″ 165 4.12 4.80 1920 960 69.9 3.50 292.6 17.8 0.89 3.50″13.98″ 165 4.12 4.80 1920 1920 139.8 6.99 585.2 35.5 1.78 6.99″ 27.96″165 4.12 4.80 1920 GoPro H3 15 0.3 0.01 4.6 0.1 0.00 0.01″ 0.06″ 653 216.17 1920 30 0.6 0.03 9.1 0.1 0.01 0.03″ 0.1″ 653 21 6.17 1920 60 1.10.06 18.3 0.3 0.01 0.06″ 0.2″ 653 21 6.17 1920 120 2.2 0.11 36.6 0.60.03 0.11″ 0.4″ 653 21 6.17 1920 240 4.4 0.22 73.1 1.1 0.06 0.22″ 0.9″653 21 6.17 1920 480 8.8 0.44 146.3 2.2 0.11 0.44″ 1.8″ 653 21 6.17 1920960 17.6 0.88 292.6 4.5 0.22 0.88″ 3.5″ 653 21 6.17 1920 1920 35.3 1.76585.2 9.0 0.45 1.76″ 7.1″ 653 21 6.17 1920

From Table 3, it is apparent that error in the measurement will berelative to the size of the object of interest (or specific dimensionsof interest within the object), with the measurement derived for smallerobjects being more accurate relative to the actual dimensions of theobject when the image-capture device is closer to the object.

As an example of measurement accuracy attainable with the inventivemethodology herein, an “estimation level of accuracy” may be appropriatewhen only approximate measurements are required to determine the amountof materials needed for a project. In some embodiments, such “estimationlevels of accuracy” are equal to or less than about 20%, or in otherembodiments, about 15% or in yet other embodiments, about 10% or morethan about 5% of the actual dimensions of the at least one object ofinterest. To illustrate, an extracted measurement value of the at leastone object of interest that is one-hundred ten (110) inches (279.4 cm)is within an “estimation level of accuracy” when the actual measurementof the at least one object of interest is 100 inches (254 cm), such thatthe error is 10%. Situations where such “estimation level of accuracy”would be valuable, for example, are to estimate the materials needed tofor carpet, wallpaper, paint, sod, roofing and the like.

In some circumstances, a better than “estimation level of accuracy” willbe appropriate. Such cases will call for a “fabrication level ofaccuracy.” In some embodiments, such “fabrication level of accuracy”means that the extracted measurement value is less than about 5%, or inother embodiments, less than about 3% or in still other embodiments,less than about 2% or less than about 1% of the actual dimensions of theat least one object of interest. Situations where such “fabricationlevel of accuracy” would be appropriate include, for example,measurements used to manufacture custom cabinets, off-site preparationof construction details (trim), identification of exact dimensions ofcomponentry (e.g., space available for appliances, BIM) and the like.

In some embodiments, software associated with the methods and devices ofthe present invention is configured to provide information regarding theerror in the measurement presented. For example, in some embodiments,when the measurement of an object is reported to the user as 10 feet(3.048 meters) along one dimension, information about any error in suchmeasurement pixel accuracy or % error is provided as set out elsewhereherein.

In a further aspect, in some embodiments, the 3D digital representationsof the at least one object of interest are derived from the plurality ofoverlapping 2D images incorporating the object(s) substantially withoutneed for manual steps to extract measurements, such as by providingmanual manipulation to extract the data necessary to generate the 3Ddigital representations. Still further, in some embodiments, themeasurements can be obtained substantially without need for a separatescaling step, such as that required to obtain measurements of objectswith the Photomodeler product, for example.

In one embodiment, an image-capture device can be integrated into amobile device to allow images of the at least one object of interest tobe obtained. Software either included in or associated with the mobiledevice can be suitably configured to allow the 2D-image processing, datageneration, generation of the 3D digital representation of the object(s)to substantially occur on the mobile device using software and hardwareassociated with the device. Such software, etc., can also be configuredto present to the user a measurement of one or more dimensions of theobject of interest or to store such measurement for use.

In one aspect, measurements of the at least one object of interest canbe obtained using a marker as a reference. For example, a ruler or otherstandard sized object can be incorporated in a scene that includes theat least one object of interest. Using the known dimensions of themarker, one or more dimensions of the object can be derived using knownmethods.

In another aspect, measurements of the at least one object of interestcan be obtained without use of, or in addition to, a marker. In thisregard, the invention utilizes an internal or “intrinsic” reference.With this intrinsic reference, the invention herein allows a user togenerate substantially accurate measurements of the at least one objectof interest. In particular, such substantially accurate measurements areprovided, in some aspects, by incorporation of the intrinsic referenceinto the software instructions associated with the image-capture deviceand/or any hardware into which the device is associated. In separateaspects, the intrinsic reference comprises one or more of: i) dimensionsgenerated from at least two focal lengths associated with theimage-capture device; ii) a library of standard object sizesincorporated in software provided to the image-capture device; iii) useridentification of a reference object in a scene that contains the atleast one object of interest; and iv) data from which measurements ofthe at least one object of interest can be derived, wherein suchmeasurement data is generated from a combination of inertial sensorsassociated with the image-capture device, where the sensors provide datacomprising: (a) an acceleration value from an accelerometer associatedwith the image-capture device; and (b) an orientation value provided bya gyroscopic sensor present the image-capture device.

With regard to an intrinsic reference derived from the focal length ofthe image-capture device most existing image-capture devices (e.g.,cameras) comprise a short depth of field, resulting in images whichappear focused only on a small 3D slice of the scene. Such features canbe utilized in the present invention to allow estimation of the depth or3D surface of an object of interest from a set of two or more imagesincorporating that object. These images can be obtained fromsubstantially the same point of view while the image-capture deviceparameters (e.g., the focal length) are modified. Using this technique,the amount of blur in captured images can be used to provide anestimation of the object depth where such depth can be used to derivemeasurements of one or more dimensions of interest of the object.

In a further aspect of the intrinsic reference feature of the presentinvention, a library of standard object identities and sizes can beincluded in the software associated with the image-capture device toprovide data from which measurement data for the at least one object ofinterest can be derived. For example, the size of one or more objectscan serve as a reference when that object appears in the same scene asthe at least one object of interest. For example, if a single togglelight switchplate, which has a standard US size of 4.5 inches (11.42 cm)in height and 2.75 inches (6.985 cm) in width, appears in a scene withan object of interest, the known standard dimensions of this switchplatecan be used as an intrinsic reference to provide a point of referencefrom which the dimensions of the object of interest can be derived. Insome aspects, the user can identify the intrinsic reference objectmanually or object recognition methodologies can be used toautomatically process the dimension data. The reference object used asthe intrinsic reference can be generated from a database of digitalphotographic and/or video images that are likely to occur in a givenenvironment, for example. In another aspect, a database of commonobjects present in a construction or contractor setting can be includedin software configurations directed toward such users. Items related tohousehold furnishings can be included in software configurationsdirected toward interior decorators. More broadly, the database mayinclude photographic and/or video images of structures within somegeneral use or location.

In a third aspect, the intrinsic reference can be provided by useridentification of an object of interest that can serve as a reference.In this regard, the software associated with the image-capture deviceand/or the hardware into which the image-capture device is integratedcan be configured to allow the user to select an object in the scene toserve as a reference, such as by way a user interface. The user canmeasure the reference object directly and input the measured value or hecan select from a library of standard objects as discussed previouslywhere such database is associated with the software of the presentinvention. For example, if the identified reference object that willserve as the intrinsic reference for providing measurement of an objectof interest present in the scene is a switchplate cover, the system 101will elicit and receive the specification of an object to be used fordimensional calibration, and the user will select the switchplate coverto serve as the intrinsic reference. The user can then measure thedimensions of the switchplate cover and input the dimensions into theappropriate fields in the user interface when that information iselicited. Calculations of the dimensions of the object of interest willthen be provided using the methodology set out elsewhere herein.Alternatively, in some embodiments, in response to the system elicitingan object to be used for dimensional calibration, the user selects theswitchplate cover as a reference object and the standard dimensions of aswitchplate cover are obtained from a library of standard object sizesincorporated within the software associated with the image-capturedevice, thereby allowing the measurements of an object of interest to beobtained as set out elsewhere herein.

In a further aspect, the intrinsic reference can be provided by sensordata obtained from inertial sensors associated with the image-capturedevice. In some embodiments, calculating the image-capture devicedisplacement between two images/frames allows resolution of scaleambiguity. In some embodiments, the image-capture device displacement isextracted from data that inertial sensors (e.g., accelerometer andgyroscope) in the image-capture device. In particular, a gyroscopemeasures orientation based on the principles of angular momentum. Anaccelerometer, on the other hand, measures gravitational andnon-gravitational acceleration. In some embodiments, integration ofinertial data generated by movement of the image-capture device overtime provides data regarding displacement that, in turn, is utilized togenerate measurements of one or more dimensions of the object ofinterest using known methods.

In some embodiments, image-capture device-specific data is obtained bysystem 101 to provide more accurate measurement of the at least oneobject of interest. To achieve such accuracy, the actual image-capturedevice specifications such as, for example, focal length, lensdistortion parameter and principal point are determined through acalibration process. In certain embodiments, a self-calibration functionis performed without image-capture device details, which can occur whensuch details are not stored. In this regard, software associated withthe image-capture device can suitably estimate information needed toprovide measurements of the at least one object of interest. In someembodiments, self-calibration of the camera is conducted using theepipolar geometry concept. Epipolar geometry between each image pair canprovide us an estimated value of the focal length. The collection ofthese estimations is used in a prediction model to predict an optimumfocal length value.

The methods, systems, devices, and software aspects of the invention canbe carried out on a wide variety of devices that can generally becategorized by the term “image-capture device.” As used herein, suchimage-capture devices in use today are integrated into mobile devicessuch as “smartphones,” mobile telephones, “tablets,” “wearable devices”(such as where a camera may be embedded or incorporated into clothing,eyeglasses or functional jewelry, etc.), laptop computers, unmannedaerial vehicles (UAVs; e.g., drones, robots), etc. Still further, theimage-capture devices 182 and 184 (see FIG. 1A) can be associated (suchas by being in communication with) desktop computers 186 and cloud-basedcomputers 181. It is contemplated by the inventors herein thatinnovations in image-capture devices will be introduced in the future.Such image-capture devices are included in the present invention ifthese devices can be configured to incorporate the inventive methodsherein.

In various aspects of the invention, all or some portion of theprocesses claimed herein can be carried out on a portable device thatincludes suitable processing capability. In recent years, there has beena proliferation of smartphones. Exemplary operating systems/smartphonesare IOS/iPhone®, Android®/Samsung Galaxy® and Windows®/Windows Phone®).As would be recognized, smartphones are wireless, compact, hand-helddevices that, in addition to basic cellular telephone functions, includea range of compact hardware. Typical smartphones have embedded (or“native”) digital cameras that include both video and staticimage-acquisition capabilities, large touchscreen displays, andbroadband or Wi-Fi capabilities allowing for the receipt andtransmission of large amounts of data to and from the Internet. Morerecently, tablet computers and wearable devices have emerged thatprovide, in pertinent part, many of the functionalities of smartphones,including image capture and processing capabilities and WiFi andcellular capabilities.

Smartphones, tablets and wearable devices not only include a range ofhardware, they are also configured to download and run a wide variety ofsoftware applications, commonly called “apps.” The proliferation ofmobile devices, with their combination of portable hardware and readilyloaded software applications, creates a platform upon which many aspectsof the invention may be practiced.

In certain aspects, the invention advantageously utilizes basic featuresof smartphones, tablets, and wearable devices, and extends thecapabilities of these devices to include accurate and convenientmeasurement of one or more objects of interest by using theimage-capture devices native on such devices. In further aspects, theprocesses described herein may convert a common smartphone, tablet,wearable device, standalone camera or the like into a measurement tool,medical device or research tool, for example. Such aspects will benefitusers by extending the functionality of these devices.

While use of multi-function smartphones, tablets, wearable devices, orthe like that incorporate image-capture devices suitably allowimplementation of the methodology herein, devices that include lessfunctionality, such as “standalone” digital cameras or video cameras,are also used in some embodiments. Such image-capture devices generallyinclude WiFi and/or cellular capabilities, as well as “apps” so as toprovide networked functionality. Accordingly, such image-capture devicescan suitably be utilized in accordance with one or more of theinventions herein. One example of a standalone digital camera that canbe used is the GoPro® H3.

In a further example, the methods herein can be performed on asingle-purpose device. For example, an image-capture device intended foruse by professionals who work with exterior and interior building spaces(e.g., architects, contractors, interior designers, etc.) can beconfigured with hardware and software suitable to allow the users toobtain measurements that they can use in their respective professionalresponsibilities. One example of such implementations is detailed in theco-assigned U.S. Provisional Patent Application No. 62/165,995 filed May24, 2015 entitled “Interior Survey Devices and Methods,” the disclosureof which is incorporated by reference in its entirety.

The methods herein can also be provided in the form of an applicationspecific integrated circuit (“ASIC”) that is customized for theparticular uses set out herein. Such ASIC can be integrated intosuitable hardware according to known methods to provide a deviceconfigured to operate the methods herein.

Still further, the present invention relates to mobile devices and thelike that are configurable to provide substantially accuratemeasurements of at least one object of interest, where such measurementsare derived from a 3D digital representation of the object of interestobtained according to the methodology herein. In one aspect, forexample, the dimensions of a roof can be obtained using a single videocamera that includes passive image-capture capability, such as thatembedded in a mobile device, thereby eliminating the need to send aperson to the location to measure the size of the roof to provide anestimate. Yet further, the dimensions of a kitchen (or, more broadly,any room or interior of a structure) can be obtained using the passiveimage-acquisition and processing methods herein thereby allowingcabinets or the like to be sized accurately without the need to send anestimator to the customer's home. Yet further, accurate dimensions of afloor area can be provided using measurement derived from distances fromwall to wall in a room so as to provide an estimate of the amount ofmaterials needed for a flooring project. As would be recognized, theability to obtain accurate measurement of locations such as roofs,kitchens, flooring and other locations would provide significantbenefits to contractors who currently must first visit a location toobtain substantially accurate measurements prior to being able toprovide a close estimation of the cost of a construction job. Suchapplications are described in the co-assigned U.S. ProvisionalApplication No. 62/165,995, previously incorporated herein.

In a further aspect, in some embodiments, the devices and methods hereinare used to provide substantially accurate measurements andcharacteristics of a person's body so as to allow custom clothing to beprepared for him or her without the need to visit a tailor. In someembodiments, such accurate body measurement are used to facilitatetelemedicine applications.

Yet further, in some embodiments, the invention herein provides accuratemeasurement of wound size and other characteristics present on a humanor an animal. Accordingly, the present invention further relates tomedical devices configured with image-capture devices and associatedsoftware that provide the disclosed benefits and features.

In further embodiments, the accurate 3D digital representations of theobject(s) can be used create accurate 3D models of the object ofinterest, where such 3D models can be generated using 3D printingdevices, etc.

In some embodiments, the methodology herein is utilized in conjunctionwith navigation utilized for robots, unmanned autonomous vehicles andthe like where such navigation utilizes image-capture devices therein.In one example, the present invention can be incorporated withSimultaneous Localization And Mapping (“SLAM”). As would be recognized,SLAM is a method that used in robotic navigation where a robot orautonomous vehicle estimates its location relative to its environment,while simultaneously avoiding any dangerous obstacles. The autonomousvehicle makes observations of surrounding landmarks from poses obtainedfrom one or more image-capture devices associated with the vehicle andprobabilistic methods are used to achieve maximum likelihood estimationof the camera trajectory and 3D structure. Although many researchefforts have been undertaken on this topic in the robotics andcomputer-vision communities, at this time no conventional methodologycan suitably provide a substantially accurate dense 3D mapping oflarge-scale environments because the focus of existing methodologies isprimarily directed towards accurate estimation of camera trajectory.

Referring to FIG. 3 and with regard to use of the methodology hereinwith robotic navigation and the like, a video stream 300 is provided tomethod 301. These images can be derived from a video image-capturedevice (such as camera 184 of FIG. 1A) as discussed elsewhere herein. Insome embodiments, line segments and corner points are detected andtracked in block 305 and block 310, respectively, of method 301. Theoutput of the detection and matching process of block 305 and block 310includes corresponding line tracks 315 and corresponding point tracks320. In block 325 SLAM is conducted as set out in more detail in thedescription of FIG. 4. In some embodiments, an initial estimation ofstructure and motion data resulting from block 325 is determined basedon the recovered structure data in block 330. In some embodiments,hybrid bundle-adjustment techniques 335 are used to furtherrefine/optimize the 3D structure and motion data 340 and are used, insome embodiments, in process 345 to generate a 3D point, line and/oredge cloud 350 representative of the at least one object of interest. Insome embodiments, 3D structure and motion data 340 are used in a 3Dplane detection process 355 to detect 3D planes 360. In someembodiments, the 3D point, line and/or edge cloud 350 and 3D planes 360are included in intelligent data smoothing in 365 to generate a 3Ddigital representation 370 incorporating the at least one object ofinterest.

Referring to FIG. 4, in some embodiments, SLAM 325 of FIG. 3 isimplemented using method 401. In some embodiments, a properimage-capture device (e.g., camera) motion model 400 is initiallyidentified. In some embodiments, such selection is as simple as a modelthat represents constant (or substantially constant) directional andangular velocity, or in other embodiments, it is more complex. Once themotion model is identified in block 400, video frames 405 are read oneby one. For each new video frame 405, an initial estimation of thecamera pose is calculated according to predictions from the cameramotion model selected in block 400. Previously detected features aretracked according to visibility constraints and new features aredetected, if necessary. In block 410, each new feature is parameterizedusing inverse depth. The feature-tracking information combined with thepredicted motion in block 415 to allow determination of future locationsin block 420. Once these locations are determined, in block 425 thepredicted camera pose and 3D structure are refined based on the newobservations. These observations are also used to update the cameramotion model in block 430. A parallax is calculated for each feature inblock 435 according to the updated parameters and if a suitable parallaxis observed, the Euclidean representation is used to replace inversedepth parameterization. A semi global optimization is then applied basedon the visibility information to find the maximum likelihood estimationof the camera poses and 3D structure in block 440. This process isrepeated until all video frames are determined by block 445 to have beenprocessed, and method 401 provides the initial estimation of structureand motion of block 330 (referring again to FIG. 3).

In conjunction with the methods herein, in some embodiments, thesoftware associated with the image-capture device and/or the hardwareinto which the image-capture device is integrated is configured toprovide the user with interactive feedback with regard to theimage-acquisition parameters. For example, in some embodiments, suchinteractive feedback provides information regarding the object ofinterest including whether the tracking is suitable to obtain aplurality of overlapping 2D images necessary to provide suitable imagesfrom which 3D digital representations of the object(s) of interest canbe generated to provide substantially accurate measurements or otheruseful information relating to the object. In some embodiments, suchprocessing is conducted in the image-capture device itself (e.g., device182 or device 184 of FIG. 1A) or the hardware in which the device isintegrated (e.g., smartphone, wearable device, etc.). In otherembodiments, the processing is performed “in the cloud” on a server 181that is in communication with the image-capture device/hardware. Inother embodiments, the processing is performed on any device (e.g.,device 186 of FIG. 1A) in communication with the image-capture deviceand/or hardware. In some embodiments, such processing is performed onboth the device/hardware and an associated server, where decision-makingregarding the location of various parts of the processing may depend onthe speed and quality that the user needs results. Yet further, in someembodiments, user feedback is provided in real time, in near real timeor on a delayed basis.

Yet further, in some embodiments, the user display of the 3D digitalrepresentation thereof is configured to provide user generated inputs tofacilitate generation of the plurality of overlapping 2D images of theat least one object of interest, the 3D digital representations of theobject(s) of interest and/or the extracted measurement values. In someembodiments, such user generated inputs include, for example, the levelof detail, a close-up of a portion of the point cloud/image, optionalcolorization, a desirable level dimension detail, etc.

In a further aspect, the software associated with the image-capturedevices and methods herein is configured to provide an accuracy valuefor the 3D digital representations of the object(s). By reporting alevel of accuracy (where such accuracy is derivable as set out elsewhereherein), a user will obtain knowledge about accuracy of the extractedmeasurement or other dimensional value of the at least one object ofinterest.

In some embodiments, the software associated with the image-capturedevices and/or hardware in which the image-capture device is integratedis configured to elicit and receive from the user a selection of aregion/area of interest in a captured image(s) of the object ofinterest. For example, in some embodiments, when a scene in which anobject of interest is captured, the software elicits and receivesselection of a specific object appearing in the scene. In an exemplaryconfiguration of such an implementation, the scene presented to the userthrough a viewfinder or screen on the image-capture device elicits andreceives the selection of an object present in the scene such as bytouch or other type of method. The object of interest can be identifiedor selected by a computer or a user. In some embodiments, the identifiedobject is then analyzed in accordance with the methods herein so as toprovide an accurate 3D digital representation of the object(s).

In some embodiments, the methods of the present invention are suitablefor use, and are performed, “in the cloud” (i.e., the software executeson server computers connected to the internet and leased on an as-neededbasis). (Note that the word “cloud” as used in the terms “point cloud”described as part of the invention is independent of, and unrelated to,“cloud computing” as such.) As would be recognized, cloud computing hasemerged as one optimization of traditional data processingmethodologies. A computing cloud is defined as a set of resources (e.g.,processing, storage, or other resources) available through a networkthat can serve at least some traditional datacenter functions for anenterprise. A computing cloud often involves a layer of abstraction suchthat the applications and users of the computing cloud may not know thespecific hardware that the applications are running on, where thehardware is located, and so forth. This allows the computing cloudoperator some additional freedom in terms of implementing resources intoand out of service, maintenance, and so on. Computing clouds may includepublic computing clouds, such as Microsoft® Azure, Amazon® Web Services,and others, as well as private computing clouds.

Communication media appropriate for use in or with the inventions of thepresent invention may be exemplified by computer-readable instructions,data structures, program modules, or other data stored on non-transientcomputer-readable media, and may include any information-delivery media.The instructions and data structures stored on the non-transientcomputer-readable media may be transmitted as a modulated data signal tothe computer or server on which the computer-implemented methods of thepresent invention are executed. A “modulated data signal” may be asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media may include wired media such asa wired network or direct-wired connection, and wireless media such asacoustic, radio frequency (RF), microwave, infrared (IR) and otherwireless media. The term “computer-readable media” as used herein mayinclude both local non-transient storage media and remote non-transientstorage media connected to the information processors usingcommunication media such as the internet. Non-transientcomputer-readable media do not include mere signals or modulated carrierwaves, but include the storage media that form the source for suchsignals.

In some embodiments, the present invention provides a method forgenerating a 3D digital representation of an object of interest. Thismethod includes: a) receiving a plurality of 2D digital images of ascene, wherein: the scene includes i) at least one object of interest,wherein the object of interest has a plurality of dimensions; ii) atleast a portion of the plurality of the 2D digital images of the sceneare overlapping with regard to the at least one object of interest; andiii) the plurality of 2D digital images are generated from a singlepassive image-capture device; and b) processing at least a portion ofthe plurality of overlapping 2D digital images that includes the atleast one object of interest using a 3D reconstruction process thatincorporates a structure from motion algorithm, thereby generating a 3Ddigital representation of the at least one object of interest, whereinmeasurements of one or more of the plurality of dimensions of the atleast one object of interest are obtainable from the 3D digitalrepresentation.

In some embodiments, the present invention provides a method forgenerating a 3D digital representation of an object of interest. Thismethod includes: receiving a plurality of 2D digital images of a scene,wherein: i) the scene includes at least one object of interest, whereinthe object of interest has a plurality of dimensions; ii) at least aportion of the plurality of the 2D digital images of the scene areoverlapping with regard to the at least one object of interest; and iii)the plurality of 2D digital images are generated from a single passiveimage-capture device. The method also includes processing at least aportion of the plurality of overlapping 2D digital images that includesthe at least one object of interest using a 3D reconstruction processthat incorporates a structure from motion algorithm, thereby generatinga 3D digital representation of the at least one object of interest; andcalculating measurements of one or more of the plurality of dimensionsof the at least one object of interest from the 3D digitalrepresentation. Some embodiments further include displaying the 3Ddigital representation of the at least one object of interest. Someembodiments further include calculating the plurality of dimensionmeasurements of the at least one object of interest from the 3D digitalrepresentation. In some embodiments, the single passive image-capturedevice is a video camera.

Some embodiments further include generating at least one of a 3D model,a 3D point cloud, a 3D line cloud, and a 3D edge cloud from the 3Ddigital representation, wherein each, independently, includes at leastone of the plurality of dimensions of the at least one object ofinterest. In some embodiments, the measurements are obtainablesubstantially without a separate scaling step.

Some embodiments further include selecting one or more of the pluralityof dimensions in the at least one object of interest, wherein each ofthe selected dimensions, independently, includes an actual measurementvalue; extracting measurement data from the selected dimensions; andprocessing the extracted measurement data to provide an extractedmeasurement value for each selected dimension.

In some embodiments, at least one of the selection steps isautomatically performed by a computer. In some embodiments, either orboth of the selection steps is elicited and received by a computer froma user. In some such embodiments, a pixel accuracy of each extractedmeasurement value, independently, is represented in pixel unitsaccording to the following formula:

((distance of object of interest from image-capture device)*(imagecapture device sensor size))/((image-capture deviceresolution*image-capture device focal length))

wherein “*” represents multiplication and “/” represents division.

In some such embodiments, the pixel accuracy of each extractedmeasurement value is about one pixel.

In some embodiments, each extracted measurement value of each selecteddimension is, independently, within about 5% of each correspondingactual measurement value.

Some embodiments further include generating boundary information for theat least one object of interest.

In some embodiments, the present invention provides a computerizedmethod of obtaining at least one measurement of an object of interest,This computerized method includes: a) receiving a plurality of 2D imagesof a scene from a single passive image-capture device, wherein theplurality of 2D images includes image data of at least one object ofinterest present in the scene, and at least a portion of the pluralityof 2D images of the scene are at least partially overlapping with regardto the at least one object of interest, thereby providing a plurality ofoverlapping 2D images that includes the at least one object of interest;b) generating, by the computer, a 3D representation of the at least oneobject of interest, wherein the 3D digital representation is obtainedfrom at least a portion of the 2D digital images incorporating theobject using a process incorporating a structure from motion algorithm;c) eliciting and receiving selections, made by either or both thecomputer or the user, of one or more dimensions of interest in the atleast one object of interest, wherein each dimension, independently,comprises an actual measurement value; d) extracting data, by thecomputer, from the 3D digital representation, wherein the extracted datacomprises measurement data comprising information corresponding to eachidentified dimension; and e) processing, by the computer, the extractedmeasurement data to provide an extracted measurement value for eachselected dimension.

In some embodiments, an accuracy of each extracted measurement value,independently, is represented in pixels according to formula:

((distance of object of interest from image-capturedevice)*(image-capture device sensor size))/((image-capture deviceresolution*image-capture device focal length)).

In some embodiments, a pixel accuracy of each extracted measurementvalue is about one pixel.

In some embodiments, the plurality of 2D images includes video images.

Some embodiments further include generating boundary information for theat least one object of interest.

In some embodiments, the present invention provides a computerizedmethod of boundary detection. In some embodiments, this method includes:a) receiving a plurality of 2D digital images of a scene, wherein: i)the scene includes at least one object of interest having a plurality ofboundaries; ii) at least a portion of the plurality of 2D digital imagesis overlapping with regard to the at least one object of interest; iii)the plurality of 2D digital images are generated from single passiveimage-capture device; and b) processing at least a portion of theplurality of overlapping 2D digital images that include the at least oneobject of interest using a method that incorporates a structure frommotion algorithm, thereby providing detected boundary information for atleast a portion of the at least one object of interest, wherein thedetected boundary information can be represented as at least one of: a3D digital representation, a 3D model, a 3D point cloud, a 3D linecloud, and a 3D edge cloud, each corresponding to at least a portion ofthe at least one object of interest. In some such embodiments, thesingle passive image-capture device is a video camera. In someembodiments, the measurements of at least a portion of the at least oneobject of interest are obtainable from the detected boundaryinformation.

At this time, there is little distinction left between hardware andsoftware implementations of aspects of systems; the use of hardware orsoftware is generally (but not always, in that in certain contexts thechoice between hardware and software can become significant) a designchoice representing cost vs. efficiency tradeoffs. There are variousinformation-processing vehicles by which processes and/or systems and/orother technologies described herein may be implemented, e.g., hardware,software, and/or firmware, and that the preferred vehicle may vary withthe context in which the processes and/or systems and/or othertechnologies are deployed. For example, if an implementer determinesthat speed and accuracy are paramount, the implementer may opt for amainly hardware and/or firmware vehicle; if flexibility is paramount,the implementer may opt for a mainly software implementation; or, yetagain alternatively, the implementer may opt for some combination ofhardware, software, and/or firmware.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes for system configuration via the use ofblock diagrams, flowcharts, and/or examples. Insofar as such blockdiagrams, flowcharts, and/or examples contain one or more functionsand/or operations, it will be understood by those within the art thateach function and/or operation within such block diagrams, flowcharts,or examples can be implemented, individually and/or collectively, by awide range of hardware, software, firmware, or virtually any combinationthereof. In one embodiment, several portions of the subject matterdescribed herein may be implemented via Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signalprocessors (DSPs), or other integrated formats. However, those skilledin the art will recognize that some aspects of the embodiments disclosedherein, in whole or in part, can be equivalently implemented inintegrated circuits, as one or more computer programs running on one ormore computers, e.g., as one or more programs running on one or morecomputer systems, as one or more programs running on one or moreprocessors, e.g., as one or more programs running on one or moremicroprocessors, as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure. In addition, those skilled inthe art will appreciate that the mechanisms of the subject matterdescribed herein are capable of being distributed as a program productin a variety of forms, and that an illustrative embodiment of thesubject matter described herein applies regardless of the particulartype of signal bearing medium used to actually carry out thedistribution. Examples of a signal-bearing medium include, but are notlimited to, the following: a recordable type medium such as a floppydisk, a hard disk drive, a CD, a DVD, a digital tape, a computer memory,etc.; and a remote non-transitory storage medium accessed using atransmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link, etc.), for example aserver accessed via the internet.

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data-processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors, e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities. A typical data processing systemmay be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein-described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

As described above, the exemplary embodiments have been described andillustrated in the drawings and the specification. The exemplaryembodiments were chosen and described in order to explain certainprinciples of the invention and their practical application, to therebyenable others skilled in the art to make and utilize various exemplaryembodiments of the present invention, as well as various alternativesand modifications thereof. As is evident from the foregoing description,certain aspects of the present invention are not limited by theparticular details of the examples illustrated herein, and it istherefore contemplated that other modifications and applications, orequivalents thereof, will occur to those skilled in the art. Manychanges, modifications, variations and other uses and applications ofthe present construction will, however, become apparent to those skilledin the art after considering the specification and the accompanyingdrawings. All such changes, modifications, variations and other uses andapplications which do not depart from the spirit and scope of theinvention are deemed to be covered by the invention which is limitedonly by the claims which follow.

1) A method for generating a 3D digital representation of an object ofinterest, the method comprising: a) receiving, into a computer, aplurality of 2D digital images of a scene, wherein: i) the sceneincludes a first object of interest, wherein the object of interest hasa plurality of dimensions; ii) at least a portion of the plurality ofthe 2D digital images of the scene are overlapping with regard to thefirst object of interest; and iii) the plurality of 2D digital imagesare generated from a single passive image-capture device; b) processing,by the computer, at least a portion of the plurality of overlapping 2Ddigital images that includes the first object of interest using a 3Dreconstruction process that incorporates a structure from motionalgorithm, thereby generating a 3D digital representation of the firstobject of interest; and c) generating, using the computer, measurementsof a first plurality of the plurality of dimensions of the first objectof interest from the 3D digital representation. 2) The method of claim1, wherein the single passive image-capture device is a video camera. 3)The method of claim 1, further comprising: using the 3D digitalrepresentation for generating at least one of a 3D model, a 3D pointcloud, a 3D line cloud, and a 3D edge cloud, wherein each,independently, comprises at least one of the plurality of dimensions ofthe first object of interest. 4) The method of claim 1, wherein theobtaining of the measurements is performed substantially without aseparate scaling operation. 5) The method of claim 1, furthercomprising: a) selecting at least one of the plurality of dimensions inthe first object of interest, wherein each of the selected dimensions,independently, comprises an actual measurement value; b) extractingmeasurement data from the selected dimensions; and c) processing theextracted measurement data to provide an extracted measurement value foreach selected dimension. 6) The method of claim 5, wherein the selectingof the at least one of the plurality of dimensions is performedautomatically by a computer. 7) The method of claim 5, wherein theselecting of the at least one of the plurality of dimensions includeseliciting and receiving into a computer information that specifies theat least one of the plurality of dimensions from a user. 8) The methodof claim 5, wherein a pixel accuracy of each extracted measurementvalue, independently, is represented in pixel units according toformula:((distance of object of interest from image-capture device)*(imagecapture device sensor size))/((image-capture deviceresolution*image-capture device focal length)). 9) The method of claim8, wherein the pixel accuracy of each extracted measurement value isabout one pixel. 10) The method of claim 5, wherein each value of theextracted measurement data of each selected dimension is, independently,within about 5% of each corresponding actual measurement value. 11) Themethod of claim 1, further comprising generating boundary informationfor the first object of interest. 12) A computerized method of obtainingat least one measurement of an object of interest comprising: a)receiving a plurality of 2D images of a scene from a single passiveimage-capture device, wherein the plurality of 2D images includes imagedata of a first object of interest present in the scene, and at least aportion of the plurality of 2D images of the scene are at leastpartially overlapping with regard to the first object of interest,thereby providing a plurality of overlapping 2D images that includes thefirst object of interest; b) generating, by the computer, a 3Drepresentation of the first object of interest, wherein the 3D digitalrepresentation is obtained from at least a portion of the 2D digitalimages incorporating the first object using a process incorporating astructure from motion algorithm; c) eliciting and receiving, from eitheror both the computer or the user, selection-identification informationthat identifies a plurality of dimensions of interest in the firstobject of interest, wherein each dimension, independently, comprises anactual measurement value; d) extracting data, by the computer, from the3D digital representation, wherein the extracted data comprisesmeasurement data comprising information corresponding to each identifieddimension; and e) processing, by the computer, the extracted measurementdata to provide an extracted measurement value for each selecteddimension. 13) The method of claim 12, wherein an accuracy of eachextracted measurement value, independently, is represented in pixelsaccording to formula:((distance of object of interest from image-capturedevice)*(image-capture device sensor size))/((image-capture deviceresolution*image-capture device focal length)). 14) The method of claim13, wherein a pixel accuracy of each extracted measurement value isabout one pixel. 15) The method of claim 12, wherein the plurality of 2Dimages are video images. 16) The method of claim 12, further comprisinggenerating boundary information for the first object of interest. 17) Amethod of boundary detection, comprising: a) receiving a plurality of 2Ddigital images of a scene, wherein: i) the scene includes a first objectof interest having a plurality of boundaries, ii) at least a portion ofthe plurality of 2D digital images is overlapping with regard to thefirst object of interest, and iii) the plurality of 2D digital imagesare generated from single passive image-capture device; and b)processing at least a portion of the plurality of overlapping 2D digitalimages that include the first object of interest using a method thatincorporates a structure from motion algorithm, thereby providingdetected boundary information for at least a portion of the first objectof interest, wherein the detected boundary information can berepresented as at least one of: i) a 3D digital representation, ii) a 3Dmodel, iii) a 3D point cloud, iv) a 3D line cloud, and v) a 3D edgecloud. 18) The method of claim 17, wherein the single passiveimage-capture device is a video camera. 19) The method of claim 17,wherein the measurements of at least a portion of the first object ofinterest are obtainable from the detected boundary information. 20) Themethod of claim 17, further comprising: c) processing, by a computer, atleast a portion of the plurality of overlapping 2D digital images thatincludes the first object of interest using a 3D reconstruction processthat incorporates a structure-from-motion algorithm, thereby generatinga 3D digital representation of the first object of interest; d)generating, using the computer, measurements of a first plurality of theplurality of dimensions of the first object of interest from the 3Ddigital representation; e) eliciting and receiving, from either or boththe computer or the user, selection-identification information thatidentifies a plurality of dimensions of interest in the first object ofinterest, wherein each dimension, independently, comprises an actualmeasurement value; f) extracting data, by the computer, from the 3Ddigital representation, wherein the extracted data comprises measurementdata comprising information corresponding to each identified dimension;and g) processing, by the computer, the extracted measurement data toprovide an extracted measurement value for each selected dimension.